< Back
New-AzureRmDataFactoryDataset
Post
NAME New-AzureRmDataFactoryDataset
SYNOPSIS
Creates a dataset in Data Factory.
SYNTAX
New-AzureRmDataFactoryDataset [-DataFactory] <PSDataFactory> [[-Name] <String>] [-File] <String> [-DefaultProfile <IAzureContextContainer>]
[-Force] [-Confirm] [-WhatIf] [<CommonParameters>]
New-AzureRmDataFactoryDataset [-ResourceGroupName] <String> [-DataFactoryName] <String> [[-Name] <String>] [-File] <String> [-DefaultProfile
<IAzureContextContainer>] [-Force] [-Confirm] [-WhatIf] [<CommonParameters>]
DESCRIPTION
The New-AzureRmDataFactoryDataset cmdlet creates a dataset in Azure Data Factory. If you specify a name for a dataset that already exists, this
cmdlet prompts you for confirmation before it replaces the dataset. If you specify the Force parameter, the cmdlet replaces the existing dataset
without confirmation.
Perform these operations in the following order:
- Create a data factory.
- Create linked services.
- Create datasets.
- Create a pipeline.
If a dataset with the same name already exists in the data factory, this cmdlet prompts you to confirm whether to overwrite the existing dataset
with the new dataset. If you confirm to overwrite the existing dataset, the dataset definition is also replaced.
PARAMETERS
-DataFactory <PSDataFactory>
Specifies a PSDataFactory object. This cmdlet creates a dataset in the data factory that this parameter specifies.
Required? true
Position? 0
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-DataFactoryName <String>
Specifies the name of a data factory. This cmdlet creates a dataset in the data factory that this parameter specifies.
Required? true
Position? 1
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-DefaultProfile <IAzureContextContainer>
The credentials, account, tenant, and subscription used for communication with azure
Required? false
Position? named
Default value None
Accept pipeline input? False
Accept wildcard characters? false
-File <String>
Specifies the full path of the JavaScript Object Notation (JSON) file that contains the description of the dataset.
Required? true
Position? 3
Default value None
Accept pipeline input? False
Accept wildcard characters? false
-Force [<SwitchParameter>]
Indicates that this cmdlet replaces an existing dataset without prompting you for confirmation.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
-Name <String>
Specifies the name of the dataset to create.
Required? false
Position? 2
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-ResourceGroupName <String>
Specifies the name of an Azure resource group. This cmdlet creates a dataset in the group that this parameter specifies.
Required? true
Position? 0
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-Confirm [<SwitchParameter>]
Prompts you for confirmation before running the cmdlet.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
-WhatIf [<SwitchParameter>]
Shows what would happen if the cmdlet runs. The cmdlet is not run.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer, PipelineVariable, and OutVariable. For more information, see
about_CommonParameters (http://go.microsoft.com/fwlink/?LinkID=113216).
INPUTS
None
This cmdlet does not accept any input.
OUTPUTS
Microsoft.WindowsAzure.Commands.Utilities.PSDataset
NOTES
* Keywords: azure, azurerm, arm, resource, management, manager, data, factories
Example 1: Create a dataset
PS C:\\>New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
DatasetName : DAWikipediaClickEvents
ResourceGroupName : ADF
DataFactoryName : WikiADF
Availability : Microsoft.DataFactories.Availability
Location : Microsoft.DataFactories.AzureBlobLocation
Policy : Microsoft.DataFactories.Policy
Structure : {}
This command creates a dataset named DA_WikipediaClickEvents in the data factory named WikiADF. The command bases the dataset on information in
the DAWikipediaClickEvents.json file.
Example 2: View availability for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Availability
AnchorDateTime :
Frequency : Hour
Interval : 1
Offset :
WaitOnExternal : Microsoft.DataFactories.WaitOnExternal
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command uses standard dot notation to display details about the Availability property of the dataset.
Example 3: View location for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Location
BlobPath : wikidatagateway/wikisampledatain/
FilenamePrefix :
Format :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy : {}
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command displays details about the Location property of the dataset.
Example 4: View validation rules for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Policy.Validation | Format-List $dataset.Location
BlobPath : wikidatagateway/wikisampledatain/
FilenamePrefix :
Format :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy : {}
MinimumRows :
MinimumSizeMB : 1
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command gets details about the validation rules for the dataset, and then passes them to the Format-List cmdlet by using the pipeline
operator. That Windows PowerShell cmdlet formats the results. For more information, type `Get-Help Format-List`.
RELATED LINKS
Online Version: https://docs.microsoft.com/en-us/powers ... orydataset
Get-AzureRmDataFactoryDataset
Remove-AzureRmDataFactoryDataset
SYNOPSIS
Creates a dataset in Data Factory.
SYNTAX
New-AzureRmDataFactoryDataset [-DataFactory] <PSDataFactory> [[-Name] <String>] [-File] <String> [-DefaultProfile <IAzureContextContainer>]
[-Force] [-Confirm] [-WhatIf] [<CommonParameters>]
New-AzureRmDataFactoryDataset [-ResourceGroupName] <String> [-DataFactoryName] <String> [[-Name] <String>] [-File] <String> [-DefaultProfile
<IAzureContextContainer>] [-Force] [-Confirm] [-WhatIf] [<CommonParameters>]
DESCRIPTION
The New-AzureRmDataFactoryDataset cmdlet creates a dataset in Azure Data Factory. If you specify a name for a dataset that already exists, this
cmdlet prompts you for confirmation before it replaces the dataset. If you specify the Force parameter, the cmdlet replaces the existing dataset
without confirmation.
Perform these operations in the following order:
- Create a data factory.
- Create linked services.
- Create datasets.
- Create a pipeline.
If a dataset with the same name already exists in the data factory, this cmdlet prompts you to confirm whether to overwrite the existing dataset
with the new dataset. If you confirm to overwrite the existing dataset, the dataset definition is also replaced.
PARAMETERS
-DataFactory <PSDataFactory>
Specifies a PSDataFactory object. This cmdlet creates a dataset in the data factory that this parameter specifies.
Required? true
Position? 0
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-DataFactoryName <String>
Specifies the name of a data factory. This cmdlet creates a dataset in the data factory that this parameter specifies.
Required? true
Position? 1
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-DefaultProfile <IAzureContextContainer>
The credentials, account, tenant, and subscription used for communication with azure
Required? false
Position? named
Default value None
Accept pipeline input? False
Accept wildcard characters? false
-File <String>
Specifies the full path of the JavaScript Object Notation (JSON) file that contains the description of the dataset.
Required? true
Position? 3
Default value None
Accept pipeline input? False
Accept wildcard characters? false
-Force [<SwitchParameter>]
Indicates that this cmdlet replaces an existing dataset without prompting you for confirmation.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
-Name <String>
Specifies the name of the dataset to create.
Required? false
Position? 2
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-ResourceGroupName <String>
Specifies the name of an Azure resource group. This cmdlet creates a dataset in the group that this parameter specifies.
Required? true
Position? 0
Default value None
Accept pipeline input? True (ByPropertyName)
Accept wildcard characters? false
-Confirm [<SwitchParameter>]
Prompts you for confirmation before running the cmdlet.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
-WhatIf [<SwitchParameter>]
Shows what would happen if the cmdlet runs. The cmdlet is not run.
Required? false
Position? named
Default value False
Accept pipeline input? False
Accept wildcard characters? false
<CommonParameters>
This cmdlet supports the common parameters: Verbose, Debug,
ErrorAction, ErrorVariable, WarningAction, WarningVariable,
OutBuffer, PipelineVariable, and OutVariable. For more information, see
about_CommonParameters (http://go.microsoft.com/fwlink/?LinkID=113216).
INPUTS
None
This cmdlet does not accept any input.
OUTPUTS
Microsoft.WindowsAzure.Commands.Utilities.PSDataset
NOTES
* Keywords: azure, azurerm, arm, resource, management, manager, data, factories
Example 1: Create a dataset
PS C:\\>New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
DatasetName : DAWikipediaClickEvents
ResourceGroupName : ADF
DataFactoryName : WikiADF
Availability : Microsoft.DataFactories.Availability
Location : Microsoft.DataFactories.AzureBlobLocation
Policy : Microsoft.DataFactories.Policy
Structure : {}
This command creates a dataset named DA_WikipediaClickEvents in the data factory named WikiADF. The command bases the dataset on information in
the DAWikipediaClickEvents.json file.
Example 2: View availability for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Availability
AnchorDateTime :
Frequency : Hour
Interval : 1
Offset :
WaitOnExternal : Microsoft.DataFactories.WaitOnExternal
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command uses standard dot notation to display details about the Availability property of the dataset.
Example 3: View location for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Location
BlobPath : wikidatagateway/wikisampledatain/
FilenamePrefix :
Format :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy : {}
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command displays details about the Location property of the dataset.
Example 4: View validation rules for a new dataset
PS C:\\>$Dataset = New-AzureRmDataFactoryDataset -ResourceGroupName "ADF" -DataFactoryName "WikiADF" -Name "DAWikipediaClickEvents" -File
"C:\\\\samples\\\\WikiSample\\\\DA_WikipediaClickEvents.json"
PS C:\\> $Dataset.Policy.Validation | Format-List $dataset.Location
BlobPath : wikidatagateway/wikisampledatain/
FilenamePrefix :
Format :
LinkedServiceName : LinkedServiceWikipediaClickEvents
PartitionBy : {}
MinimumRows :
MinimumSizeMB : 1
The first command creates a dataset named DA_WikipediaClickEvents, as in a previous example, and then assigns that dataset to the $Dataset
variable.
The second command gets details about the validation rules for the dataset, and then passes them to the Format-List cmdlet by using the pipeline
operator. That Windows PowerShell cmdlet formats the results. For more information, type `Get-Help Format-List`.
RELATED LINKS
Online Version: https://docs.microsoft.com/en-us/powers ... orydataset
Get-AzureRmDataFactoryDataset
Remove-AzureRmDataFactoryDataset