Extracting data from a complex MongoDB database using PyMongo and converting it to a .csv file


Extracting data from a complex MongoDB database using PyMongo and converting it to a .csv file



I have a complex MongoDB database, consisting of documents nested upto 7 levels deep. I need to use PyMongo to extract the data, and then convert the extracted data to a .csv file.





What have you tried so far?
– Adam Smith
Jul 2 at 4:08





Can you use mongoexport?
– Astro
Jul 2 at 4:11





So far I am able to extract the entire database and store it as a Python object. I am then able to convert this object to a .csv file. However the .csv file has thousands of columns. I need to know how I can extract the data in a clean manner.
– pack24
Jul 2 at 4:11





@Astro I can use mongoexport, but the .csv file has thousands of columns. I need to extract the data in an organized manner. I'm ok with extracting the data in multiple csv files and then combining all csv files into one. I'm not sure how to proceed with that though; I just know how to extract data as a whole.
– pack24
Jul 2 at 4:13




1 Answer
1



You can try using json_normalize.
It is used to flatten the json.Reads data to a dataframe which can be stored in csv later.



For eg:


from pandas.io.json import json_normalize

# mongo_value is your mongo query
mongo_aggregate = db.events.aggregate(mongo_value)

mongo_df = json_normalize(list(mongo_aggregate))

# print(mongo_df)
mongo_columns = list(mongo_df.columns.values)

#just picks the column_name instead of properties.something.something.column_name

for w in range(len(mongo_columns)):
mongo_columns[w] = mongo_columns[w].split('.')[-1].lower()

mongo_df.columns = mongo_columns



For reference read this https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.io.json.json_normalize.html





Thank you for your help!
– pack24
Jul 2 at 12:28






By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

-

Popular posts from this blog

api-platform.com Unable to generate an IRI for the item of type

api-platform.com Unable to generate an IRI for the item of type I'm trying to setup my first api-platform instance and have run into some presumably noob questions around IDE. What i'm trying to do is setting up an register and reset entity in order to deal with registrations and password reminders from an app. And then have the entity's passed to a custom controller to do some checks, balances and mails afterwards (haven’t gotten around to that yet). But I keep running into this error when posting to my custom entities: { "@context": "/api/contexts/Error", "@type": "hydra:Error", "hydra:title": "An error occurred", "hydra:description": "Unable to generate an IRI for the item of type "AppXXXUserBundleEntityRegister"", "trace": [ { "namespace": "", "short_class": "", "c
Read more

How to set up datasource with Spring for HikariCP?

How to set up datasource with Spring for HikariCP? Hi I'm trying to use HikariCP with Spring for connection pool. I'm using jdbcTempLate and JdbcdaoSupport. This is my spring configuration file for datasource: <bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource"> <property name="dataSourceClassName" value="oracle.jdbc.driver.OracleDriver"/> <property name="dataSource.url" value="jdbc:oracle:thin:@localhost:1521:XE"/> <property name="dataSource.user" value="username"/> <property name="dataSource.password" value="password"/> </bean> But unfortunately the following error message is generating: Cannot resolve reference to bean 'dataSource' while setting bean property 'dataSource'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dataSource'
Read more

Display dokan vendor name on Woocommerce single product pages

Display dokan vendor name on Woocommerce single product pages With woocommerce I am using Dokan plugin and I am trying to display the vendor name, rating and vendor location on single product pages. I tried this code to display vendor name but no luck: //show store name add_action( 'woocommerce_single_product_summary', 'show_store_name', 20 ); function show_store_name() { printf( '<b>Seller Name:</b> <a href="%s">%s</a>', dokan_get_store_url( $author->ID ), $store_info['store_name'] ); } Any help is appreciated. 1 Answer 1 I don't use dokan plugin. Here is the way to get the (post) author ID and to make your code work: add_action( 'woocommerce_single_product_summary', 'display_author_name', 20 ); function display_author_name() { // Get the author ID (the vendor ID) $vendor_id = get_post_field( &#
Read more