Multi tool use

bq load: “BigQuery error in load operation: Not found: Project …”

I'm trying to load some ndjson data. First, creating a table works flawlessly:

> bq mk --table <project-id>:<my-dataset.newtable> newtable.schema.json
Table '<project-id>:<my-database.newtable>' successfully created.


However, the bq load command does not:

bq load

> bq load --source_format=NEWLINE_DELIMITED_JSON <project-id>:<my-dataset.newtable> gs://<project-id>.appspot.com/newtable.ndjson
BigQuery error in load operation: Not found: Project <project-friendly-name>


Please also note also:

I have no problem running the job from BigQuery's web interface.

I have set <project-id> as the default project through the bq init command, but I get the same error, even when creating a table, when I don't specify it.

<project-id>

bq init

Is there an issue with some environment variables that have not been set correctly ?

2 Answers
2

The bq load command usually follows the following structure.

bq --location=[LOCATION] load --source_format=[FORMAT] [DATASET].[TABLE] [PATH_TO_SOURCE] [SCHEMA]

As in the standard bq load command, you don't have to mention the project if you are loading data within the same project that you have logged in you cli. Also you need to mention the schema unless you have auto detect flag set in you command.

The following command allows you to identify the project that you have access to.

gcloud config list

bq mk --table [PROJECT_ID]:[DATASET].[TABLE] [SCHEMA]

gcloud config list

Ok. Interestingly and unlike with bq mk, with bq load, selecting the <project-id> with [PROJECT_ID]:[DATASET].[TABLE], or throught bq init (and the --location=[LOCATION] option with a fully-qualified Cloud Storage URI as a file) is still irrelevant.

bq mk

bq load

<project-id>

[PROJECT_ID]:[DATASET].[TABLE]

bq init

--location=[LOCATION]

I still had to run either:

gcloud config set project <project-id>

bq load --project_id=<project-id> ...

or

gcloud init

So to sum up, this works:

bq load --project_id=<project-id> --source_format=NEWLINE_DELIMITED_JSON <my-dataset.newtable> gs://<project-id>.appspot.com/newtable.ndjson


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.