scrapy Archives - ISbyR https://isbyr.com/tag/scrapy/ Infrequent Smarts by Reshetnikov Thu, 21 Apr 2022 23:46:12 +0000 en-US hourly 1 https://wordpress.org/?v=6.7.2 Format Scrapy Export https://isbyr.com/format-scrapy-export/ https://isbyr.com/format-scrapy-export/#respond Thu, 21 Apr 2022 23:46:11 +0000 http://isbyr.com/?p=823 Here is how you can format Scrapy export of your items For all of these options make sure that your spider actually yields some results in your parse method. Using a known filename extension Using one of the known file extensions that you provide in the -O option for the crawl command. These are json, … Continue reading Format Scrapy Export

The post Format Scrapy Export appeared first on ISbyR.

]]>
Here is how you can format Scrapy export of your items

For all of these options make sure that your spider actually yields some results in your parse method.

Using a known filename extension

Using one of the known file extensions that you provide in the -O option for the crawl command.

These are json, jsonlines, jl, csv, xml, marshal and pickle


Using the -t option

Provide the format in the -t option for the crawl command.

Accepted formats are the same (json, jsonlines, jl, csv, xml, marshal and pickle )

Using the FEEDS property of the spider

Provide a value to the FEEDS property in your spider

More info

Can be found in the official docs

import scrapy

class MySpider(scrapy.Spider):
    name = "myspider"
    custom_settings = {
        'FEEDS': {
            'export.jsonlines': {       # file URI (name)
                'format': 'jsonlines',  # json/jsonlines/csv/xml/marshal/pickle
            }
        }
    }

The post Format Scrapy Export appeared first on ISbyR.

]]>
https://isbyr.com/format-scrapy-export/feed/ 0