We are seeking a highly skilled Python Scrapy Expert to join our team on a project focused on web scraping and API development. The ideal candidate will possess extensive experience in scraping various websites and handling large volumes of data. Our project involves continuous scraping for price monitoring and other related tasks, requiring expertise in proxy management to prevent our scrapers from being blocked by targeted websites.
Responsibilities:
Develop and maintain web scraping scripts using Python Scrapy framework to extract data from hundreds of websites efficiently.
Implement strategies for handling large datasets and continuous scraping processes.
Utilize proxy management techniques to prevent IP blocking and ensure uninterrupted scraping operations.
Write and maintain business logic components to process scraped data effectively.
Collaborate with the team to integrate scraping functionalities into the larger project framework.
Develop APIs using Flask framework to expose scraped data for consumption by other modules or external applications.
Ensure code quality, performance, and reliability of scraping and API components.
Participate in code reviews, troubleshooting, and optimization efforts.
Adhere to Git workflows and agile methodologies for version control and project management.
Requirements:
Proficiency in Python programming language.
Extensive experience with web scraping using Scrapy framework, including handling dynamic content, pagination, and anti-scraping measures.
Strong understanding of proxy management techniques to avoid IP blocking.
Familiarity with Flask framework for API development.
Ability to write efficient and scalable code for handling large datasets.
Experience with implementing business logic in Python applications.
Proficient with Git for version control and collaboration.
Familiarity with agile development methodologies.
Excellent problem-solving skills and attention to detail.
Strong communication and collaboration skills to work effectively in a team environment.
This is a challenging role within a larger project where web scraping plays a crucial role. If you are passionate about extracting insights from data and have the expertise to overcome the challenges associated with web scraping, we encourage you to apply.
Please include relevant work samples or portfolio demonstrating your experience with Python Scrapy and Flask development when applying.
This job is already closed and no longer accepting applicants, sorry.