ウェブスクレイピング用のスクレイパーの構築方法: n8nとCapSolverを使用して

{
  "nodes": [
    {
      "parameters": {
        "content": "## Scraping \u2014 Price & Product Monitor\n\n### How it works\n\n1. Triggers either by schedule or webhook input to start price monitoring.\n2. Solves reCAPTCHA to access the targeted product page.\n3. Retrieves and extracts data from the product page for further analysis.\n4. Compares newly fetched data with previously stored data to detect changes.\n5. Determines if changes occurred and prepares alerts if needed.\n6. Sends responses based on data analysis through designated channels.\n\n### Setup steps\n\n- [ ] Configure scheduled trigger interval in 'Every 6 Hours' node.\n- [ ] Set target website details in 'Set Target Config [Schedule]'.\n- [ ] Configure reCAPTCHA solver with API key in 'Solve reCAPTCHA v3' nodes.\n- [ ] Set up Webhook URL and path in 'Webhook Trigger'.\n- [ ] Define alert criteria and destination in 'Build Alert' nodes.\n\n### Customization\n\nAdjust the extraction pattern in 'Extract Data' nodes to fit specific product details.",
        "height": 896,
        "width": 480
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -1328,
        -304
      ],
      "id": "52c7808e-d2bc-4779-85e6-909a51066338",
      "name": "Sticky Note"
    },
    {
      "parameters": {
        "content": "## Scheduled trigger setup\n\nInitializes the data monitoring process every 6 hours using a schedule trigger and sets the target configuration for scraping.",
        "height": 320,
        "width": 496,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -752,
        -160
      ],
      "id": "3c5cee67-552d-48ea-8717-7c5126269e2e",
      "name": "Sticky Note1"
    },
    {
      "parameters": {
"content": "## スケジュールされたスクリーニングプロセス\n\nスケジュールされたトリガーに従って、reCAPTCHAを解決し、製品ページを取得し、データを抽出して以前の記録と比較して変更を識別します。",
        "height": 496,
        "width": 1680,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -144,
        -304
      ],
      "id": "d0315be2-111c-4893-bf42-2f2cc2eb186f",
      "name": "Sticky Note2"
    },
    {
      "parameters": {
        "content": "## Webhookトリガーのセットアップ\n\nインバウンドWebhook経由で手動でトリガーされるデータモニタリングを処理し、進むためにreCAPTCHAを解決します。",
        "height": 304,
        "width": 816,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -768,
        336
      ],
      "id": "a78f1606-07fb-40fd-af82-e1dc9b766206",
      "name": "Sticky Note3"
    },
    {
      "parameters": {
        "content": "## Webhookスクリーニングプロセス\n\nWebhookトリガーされたリクエストを処理し、製品ページを取得し、データを抽出し、変更を判断します。",
        "height": 272,
        "width": 1088,
        "color": 7
      },
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        160,
        320
      ],
      "id": "1a677fd9-a3a8-404f-ba9a-2b087d7bfe11",
      "name": "Sticky Note4"
    },
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "field": "hours",
              "hoursInterval": 6
            }
          ]
        }
      },
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.3,
      "position": [
        -704,
        0
      ],
      "id": "sc-901",
      "name": "6時間ごと"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "cfg-001",
              "name": "websiteURL",
              "value": "https://YOUR-TARGET-SITE.com/product-page",
              "type": "string"
            },
            {
              "id": "cfg-002",
              "name": "websiteKey",
              "value": "YOUR_SITE_KEY_HERE",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        -400,
        0
      ],
      "id": "sc-900",
      "name": "Set Target Config [Schedule]"
    },
    {
      "parameters": {
        "operation": "reCAPTCHA v3",
        "websiteURL": "={{ $json.websiteURL }}",
        "websiteKey": "={{ $json.websiteKey }}",
        "optional": {
          "pageAction": "view_product"
        }
      },
      "type": "n8n-nodes-capsolver.capSolver",
      "typeVersion": 1,
      "position": [
        -96,
        0
      ],
      "id": "sc-902",
      "name": "Solve reCAPTCHA v3",
      "credentials": {
        "capSolverApi": {
          "id": "BeBFMAsySMsMGeE9",
          "name": "CapSolver account"
        }
      }
    },
    {
      "parameters": {
        "method": "POST",
        "url": "={{ $('Set Target Config [Schedule]').first().json.websiteURL }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
            },
            {
              "name": "x-recaptcha-token",
              "value": "={{ $json.data.solution.gRecaptchaResponse }}"
            }
          ]
        },
        "options": {
          "response": {
            "response": {}
          }
        }
      },
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        208,
        0
      ],
      "id": "sc-903",
      "name": "Fetch Product Page"
    },
    {
      "parameters": {
        "operation": "extractHtmlContent",
        "extractionValues": {
          "values": [
            {
              "key": "price",
              "cssSelector": ".product-price, [data-price], .price"
            },
            {
              "key": "productName",
              "cssSelector": "h1, .product-title"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.html",
      "typeVersion": 1.2,
      "position": [
        512,
        0
      ],
      "id": "sc-904",
      "name": "Extract Data"
    },
    {
      "parameters": {
        "jsCode": "const staticData = $workflow.staticData;\nconst currentPrice = $input.first().json.price;\nconst previousPrice = staticData.lastPrice;\nconst productName = $input.first().json.productName || 'Product';\nconst parsePrice = (str) => { if (!str) return null; const match = str.match(/[\\d,]+\\.?\\d*/); return match ? parseFloat(match[0].replace(',', '')) : null; };\nconst currentNum = parsePrice(currentPrice);\nconst previousNum = parsePrice(previousPrice);\nstaticData.lastPrice = currentPrice;\nstaticData.lastChecked = new Date().toISOString();\nconst changed = previousNum !== null && currentNum !== null && currentNum !== previousNum;\nconst direction = changed ? (currentNum < previousNum ? 'dropped' : 'increased') : 'unchanged';\nconst diff = changed ? Math.abs(currentNum - previousNum).toFixed(2) : '0';\nreturn [{ json: { productName, currentPrice, previousPrice: previousPrice || 'first check', changed, direction, diff: changed ? `$${diff}` : null, checkedAt: new Date().toISOString() } }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        800,
        0
      ],
      "id": "sc-905",
      "name": "Compare Data"
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict",
            "version": 2
          },
          "conditions": [
            {
              "id": "if-1",
              "leftValue": "={{ $json.changed }}",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        1104,
        0
      ],
      "id": "sc-906",
      "name": "Data Changed?"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "a1",
              "name": "alert",
              "value": "=Price {{ $json.direction }} for {{ $json.productName }}: {{ $json.previousPrice }} \u2192 {{ $json.currentPrice }}",
              "type": "string"
            },
            {
              "id": "a2",
              "name": "severity",
              "value": "={{ $json.direction === 'dropped' ? 'deal' : 'info' }}",
              "type": "string"
            },
            {
              "id": "a3",
              "name": "checkedAt",
              "value": "={{ $json.checkedAt }}",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1392,
        -192
      ],
      "id": "sc-907",
      "name": "Build Alert"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "n1",
              "name": "status",
              "value": "no_change",
              "type": "string"
            },
            {
              "id": "n2",
              "name": "currentPrice",
              "value": "={{ $json.currentPrice }}",
              "type": "string"
            },
            {
              "id": "n3",
              "name": "checkedAt",
              "value": "={{ $json.checkedAt }}",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1392,
        32
      ],
      "id": "sc-908",
      "name": "No Change"
    },
    {
      "parameters": {
        "httpMethod": "POST",
        "path": "price-monitor",
        "responseMode": "responseNode",
        "options": {}
      },
      "type": "n8n-nodes-base.webhook",
      "typeVersion": 2.1,
      "position": [
        -720,
        464
      ],
      "id": "sc-909",
      "name": "Webhook Trigger",
      "webhookId": "sc-909-webhook",
      "onError": "continueRegularOutput"
    },
    {
      "parameters": {
        "operation": "reCAPTCHA v3",
        "websiteURL": "={{ $json.body.websiteURL }}",
        "websiteKey": "={{ $json.body.websiteKey }}",
        "optional": {
          "pageAction": "={{ $json.body.pageAction || 'view_product' }}"
        }
      },
      "type": "n8n-nodes-capsolver.capSolver",
      "typeVersion": 1,
      "position": [
        -96,
        464
      ],
      "id": "sc-910",
      "name": "Solve reCAPTCHA v3 [Webhook]",
      "credentials": {
        "capSolverApi": {
          "id": "BeBFMAsySMsMGeE9",
          "name": "CapSolver account"
        }
      }
    },
    {
      "parameters": {
        "method": "POST",
        "url": "={{ $('Webhook Trigger').item.json.body.websiteURL }}",
        "sendHeaders": true,
        "headerParameters": {
          "parameters": [
            {
              "name": "user-agent",
              "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36"
            },
            {
              "name": "x-recaptcha-token",
              "value": "={{ $json.data.solution.gRecaptchaResponse }}"
            }
          ]
        },
        "options": {
          "response": {
            "response": {}
          }
        }
      },
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.3,
      "position": [
        208,
        432
      ],
      "id": "sc-911",
      "name": "Fetch Product Page [Webhook]"
    },
    {
      "parameters": {
        "operation": "extractHtmlContent",
        "extractionValues": {
          "values": [
            {
              "key": "price",
              "cssSelector": ".product-price, [data-price], .price"
            },
            {
              "key": "productName",
              "cssSelector": "h1, .product-title"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.html",
      "typeVersion": 1.2,
      "position": [
        512,
        432
      ],
      "id": "sc-912",
      "name": "Extract Data [Webhook]"
    },
    {
      "parameters": {
        "jsCode": "const staticData = $workflow.staticData;\nconst currentPrice = $input.first().json.price;\nconst previousPrice = staticData.lastPrice;\nconst productName = $input.first().json.productName || 'Product';\nconst parsePrice = (str) => { if (!str) return null; const match = str.match(/[\\d,]+\\.?\\d*/); return match ? parseFloat(match[0].replace(',', '')) : null; };\nconst currentNum = parsePrice(currentPrice);\nconst previousNum = parsePrice(previousPrice);\nstaticData.lastPrice = currentPrice;\nstaticData.lastChecked = new Date().toISOString();\nconst changed = previousNum !== null && currentNum !== null && currentNum !== previousNum;\nconst direction = changed ? (currentNum < previousNum ? 'dropped' : 'increased') : 'unchanged';\nconst diff = changed ? Math.abs(currentNum - previousNum).toFixed(2) : '0';\nreturn [{ json: { productName, currentPrice, previousPrice: previousPrice || 'first check', changed, direction, diff: changed ? `$${diff}` : null, checkedAt: new Date().toISOString() } }];"
      },
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        800,
        432
      ],
      "id": "sc-913",
      "name": "Compare Data [Webhook]"
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict",
            "version": 2
          },
          "conditions": [
            {
              "id": "if-2",
              "leftValue": "={{ $json.changed }}",
              "operator": {
                "type": "boolean",
                "operation": "true",
                "singleValue": true
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        1104,
        432
      ],
      "id": "sc-914",
      "name": "Data Changed? [Webhook]"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "a4",
              "name": "alert",
              "value": "=Price {{ $json.direction }} for {{ $json.productName }}: {{ $json.previousPrice }} \u2192 {{ $json.currentPrice }}",
              "type": "string"
            },
            {
              "id": "a5",
              "name": "severity",
              "value": "={{ $json.direction === 'dropped' ? 'deal' : 'info' }}",
              "type": "string"
            },
            {
              "id": "a6",
              "name": "checkedAt",
              "value": "={{ $json.checkedAt }}",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1424,
        384
      ],
      "id": "sc-915",
      "name": "Build Alert [Webhook]"
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "n4",
              "name": "status",
              "value": "no_change",
              "type": "string"
            },
            {
              "id": "n5",
              "name": "currentPrice",
              "value": "={{ $json.currentPrice }}",
              "type": "string"
            },
            {
              "id": "n6",
              "name": "checkedAt",
              "value": "={{ $json.checkedAt }}",
              "type": "string"
            }
          ]
        },
        "options": {}
      },
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        1440,
        592
      ],
      "id": "sc-916",
      "name": "No Change [Webhook]"
    },
    {
      "parameters": {
        "respondWith": "json",
        "responseBody": "={{ JSON.stringify($json) }}",
        "options": {}
      },
      "type": "n8n-nodes-base.respondToWebhook",
      "typeVersion": 1.5,
      "position": [
        1712,
        512
      ],
      "id": "sc-917",
      "name": "Respond to Webhook"
    }
  ],
  "connections": {
    "6時間ごと": {
      "main": [
        [
          {
"node": "ターゲット設定のスケジュールを設定",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "ターゲット設定のスケジュールを設定": {
      "main": [
        [
          {
            "node": "reCAPTCHA v3を解決",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "reCAPTCHA v3を解決": {
      "main": [
        [
          {
            "node": "製品ページを取得",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "製品ページを取得": {
      "main": [
        [
          {
            "node": "データを抽出",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データを抽出": {
      "main": [
        [
          {
            "node": "データを比較",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データを比較": {
      "main": [
        [
          {
            "node": "データが変更されましたか？",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データが変更されましたか？": {
      "main": [
        [
          {
            "node": "アラートを構築",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "変更なし",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Webhookトリガー": {
      "main": [
        [
          {
            "node": "reCAPTCHA v3 [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "reCAPTCHA v3 [Webhook]": {
      "main": [
        [
          {
            "node": "製品ページを取得 [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "製品ページを取得 [Webhook]": {
      "main": [
        [
          {
            "node": "データを抽出 [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データを抽出 [Webhook]": {
      "main": [
        [
          {
            "node": "データを比較 [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データを比較 [Webhook]": {
      "main": [
        [
          {
            "node": "データが変更されましたか？ [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "データが変更されましたか？ [Webhook]": {
      "main": [
        [
          {
            "node": "アラートを構築 [Webhook]",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "変更なし [Webhook]",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "アラートを構築 [Webhook]": {
      "main": [
        [
          {
            "node": "Webhookに応答",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "変更なし [Webhook]": {
      "main": [
        [
          {
            "node": "Webhookに応答",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "pinData": {},
  "meta": {
    "instanceId": "962ff0267b713be0344b866fa54daae28de8ed2144e2e6867da355dae193ea1f"
  }
}