...
Background processes will move the data from ingest Raw data bucket to the raw submissions buckets Standardized data bucket under a folder labelled based on the date it was uploaded. As a data provider, you will have a folder in the data lake Data Lake that contains all of the data you upload.
Local Object >> Ingest Bucket >> Raw Submissions Bucket 'Drop-zone' -> 'Raw-data' -> 'Standardized-data'
Data uploads can be verified by running the below AWS CLI command on the raw submissions Standardized data bucket to list the objects there.
| Code Block |
|---|
aws s3 ls s3://prod.sdc.dot.gov.data-lake. |
...
standardized-data/<data-provider> --profile sdc |
The Standardized data bucket name is provided in the table below the command. The “project name” and “data provider name” were are provided in the welcome email.
AWS CLI Command:
...
DEV | TEST | PROD | |
|---|---|---|---|
Standardized data bucket name |
Configuration Files
When the uploaded data reaches the Raw Submissions Bucket, it will be checked by the validation lambda function. This function confirms that all the correct fields exist for each message and that the