Previous Project
DZI Charity Scraper
This project implements a recurring data ingestion pipeline that scrapes and processes data from the DZI website to build a structured database of German charitable organizations. The data is used to power a donation interface, allowing users to direct affiliate-generated funds to a charity of their choice. The system emphasizes modularity, separating scheduling, orchestration, and scraping logic.
Kolping Foundation Rudolf-Geiselberger
+2Die Kolpingstiftung-Rudolf-Geiselberger leistet in erster Linie „Hilfe zur Selbsthilfe“. Sie dient der Förderung der Aufgaben des Kolpingwerkes, insbesondere der Völkerverständigung und Entwicklungszu...
Social WelfareCommunity DevelopmentDisaster ReliefUkraine Emergency AidInternational Understanding+6 more
Implementation
Registers and schedules the DZI importer task.
1@Component
2class DziImporterTask(
3 private val dziImporterTaskExecutor: DziImporterTaskExecutor
4) {
5 companion object {
6 private val DZI_IMPORTER_TASK = TaskDescriptor.of("dzi-importer-task", ScheduleAndPageCounter::class.java)
7 }
8
9 /**
10 * Called automatically when the Spring application context is fully initialized.
11 * Schedules the recurring task if it hasn't been scheduled yet.
12 */
13 @EventListener
14 fun setupRecurringTasks(event: ContextRefreshedEvent) {
15 val schedulerClient = event.applicationContext.getBean(SchedulerClient::class.java)
16 schedulerClient.scheduleIfNotExists(
17 DZI_IMPORTER_TASK.instance("default").data(
18 ScheduleAndPageCounter(Schedules.fixedDelay(Duration.ofSeconds(0)), 1)
19 ).scheduledAccordingToData()
20 )
21 }
22
23 /**
24 * Defines the actual recurring task and delegates its execution to the executor.
25 */
26 @Bean("dziImporterTaskBean")
27 fun dziImporterTask(): RecurringTaskWithPersistentSchedule<ScheduleAndPageCounter> =
28 Tasks.recurringWithPersistentSchedule(DZI_IMPORTER_TASK).executeStateful { taskInstance, _ ->
29 dziImporterTaskExecutor.execute(taskInstance.data)
30 }
31}
32
33
34
35