Parser va grabber yasashni o'rganamiz

3339

Yangi formatda davom etish uchun: https://uzfor.net/61211-parser-va-grabber-yasashni-organamiz.html

Parser va grabber yasashni o'rganamiz

Xabarlar

developer [323] [off]
#1 17:46:19, 5 May 2019

Assalomu alaykum do'slar forumda adashmasam parser,grabber yaratish haqidagi mavzular yo'q shu mavzuda parser va grabber yaratishni o'rganamiz.
birinchi bo'lib o'zim bilgan oddiy grabber yaratishni ko'rsatib bersam

<?php
#### grablanayotgan sayt manzili, sahifasi 
$uzg = file_get_contents('http://site.uz'.$_SERVER['QUERY_STRING']); 
echo $uzg;
?>

shu eng oddiy grabber
qirqishlar, o'zgartirishlarsiz

Grabberni asosini tashkil etuvchi funksiya bu

<?php file_get_contents();
?>

3 kun 4 soat 45 daqiqa 43 soniyadan keyin yozdi:
grabber parserchilar bormi marhamat bilimlarizni o'rtoqlashing
3 kun 10 soat 7 daqiqa 34 soniyadan keyin yozdi:
oddiy grabberlarning eng sodda 3ta funksiyasi:

<?php
#1
$uzg = file_get_contents('sayt.uz'.$_SERVER['QUERY_STRING']); /// grabberning asosiy ishlash funksiyasi grablayotgan saytingizni sizni saytingizda ochib beradi
#2
preg_match(|'olib tashlanadigan qism boshi'(.*?)'oxiri'|is,'olib tashlangan qism orniga yoziladi',$uzg); //regulyarniy virojeniya funksiyasi deyiladi
#3
str_replace(search,replace,subject);
//subject -- manba yani $uzg ga tenglab olgan qism
//search --$uzg dan qidirilayotgan qism
//replace-- searchga yozilgan so'zni o'rniga qo'yiluvchi belgi yoki so'z
?>

3 kun 10 soat 21 daqiqa 47 soniyadan keyin yozdi:
grabber yasash funksiyalari saytni ctrl+u qilib korilgandagi yani saytni skaner qilib korilgandagi html,css,js kodlari bilan ishlaydi menimcha ko'pchilikka yangi o'rganayotganlarga qirqib olish o'rniga qo'yish qiyin tuyulsa kerak chunki o'zimham qirqib olishni yaqindagina o'rgadim va sizlarga ham o'rgatishni istadim
3 kun 10 soat 37 daqiqa 18 soniyadan keyin yozdi:
sizlarga bitta sodda grabber yasab ko'rsatsam qirqib olishni tushunib olishingiz uchun

masalan siz falonchi.uz saytini skanerlab olsangiz, yoki kompda bolsangiz osha saytga kirib ctrl+u qilishning ozi kifoya sizga saytning ochiq kodlarini chiqarib beradi

masalan siz grablayotgan saytni skaner qilinganda quyidagicha kodlar chiqdi deylik

<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>Document</title>
</head>
<body>
<h3>reklama</h3>
<h3>reklama</h3>
<h1>ASSALOMU ALAYKUM</h1>
<p> Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Facere ipsam, iusto ab quas excepturi aperiam. Minima in repellat libero, perspiciatis, dolores nam, sapiente labore accusantium sed placeat iusto nisi a.Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Maiores velit, ea aliquam culpa perferendis nobis ab exercitationem possimus quam, ipsam sunt deserunt, alias itaque quaerat nihil facilis eveniet nemo beatae?Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Dignissimos modi sit corrupti asperiores, cumque id, enim recusandae reprehenderit unde totam necessitatibus qui expedita, ad. Vel nam optio quia eligendi, molestias?Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Ipsam obcaecati, dolores asperiores rerum dolorum deserunt.
<br> Ullam fugit commodi nihil nobis.
<br> Labore tempora, odio eius? Culpa aperiam esse voluptate, at ad!Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Sed fugiat, fuga iste enim dignissimos qui ullam inventore adipisci reprehenderit vitae quasi ducimus itaque maiores perferendis quis quia possimus officia illum!Lorem ipsum dolor sit amet, consectetur adipisicing elit.
Rerum, sapiente neque provident. <br>
Ex, quos ut aliquid explicabo, cupiditate numquam, iure voluptatem consectetur atque, quae perferendis nulla iste distinctio dolorum voluptas.Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Recusandae velit ducimus, consequatur id animi facilis dolor doloremque corrupti impedit odio cum, sed quam deserunt ipsam quia consequuntur perspiciatis repudiandae voluptatibus.</p>
<span>reklamalar</span>
</body>
</html>

aytaylik sizga faqat h1 teglari ichidagi va p tegi orasidagi sozlar saytingizda chiqishini hohlaysiz demak qolgan qismini kesib tashlay kerak bu preg_match() funksiyasi orqali qilinadi

demak yuqoridagi skannerlangan kodlardan keraksizlarini qirqib tashlayman

<?php
//birinchi yuqori qismdan <!DOCTYPE html> dan boshlanib </h3> tegigacha kesib tashlayman
$uzg = file_get_contents(''.$_SERVER['QUERY_STRING']);
preg_match(|'<!DOCTYPE html>'(.*?)'</h3>'|is,'',$uzg); // qirqib tashlangan belgilarni orniga hechnarsa yozmiyman men shuning uchun is dan keyin '' bosh qiymat qoydim
//va pastki qismdan kerakmas joylarni qirqib tashlayman
//span tegidan boshlab html ning yopilish tegigacha
preg_match(|'<span>'(.*?)'</html>'|is,'',$uzg);
//va garblangan malumotni ekranga chiqaramiz
echo $uzg;
?>

va bizda faqat quyidagi yozuvlar ekranga chiqadi
ASSALOMU ALAYKUM
Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Facere ipsam, iusto ab quas excepturi aperiam. Minima in repellat libero, perspiciatis, dolores nam, sapiente labore accusantium sed placeat iusto nisi a.Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Maiores velit, ea aliquam culpa perferendis nobis ab exercitationem possimus quam, ipsam sunt deserunt, alias itaque quaerat nihil facilis eveniet nemo beatae?Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Dignissimos modi sit corrupti asperiores, cumque id, enim recusandae reprehenderit unde totam necessitatibus qui expedita, ad. Vel nam optio quia eligendi, molestias?Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Ipsam obcaecati, dolores asperiores rerum dolorum deserunt.
<br> Ullam fugit commodi nihil nobis.
<br> Labore tempora, odio eius? Culpa aperiam esse voluptate, at ad!Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Sed fugiat, fuga iste enim dignissimos qui ullam inventore adipisci reprehenderit vitae quasi ducimus itaque maiores perferendis quis quia possimus officia illum!Lorem ipsum dolor sit amet, consectetur adipisicing elit.
Rerum, sapiente neque provident. <br>
Ex, quos ut aliquid explicabo, cupiditate numquam, iure voluptatem consectetur atque, quae perferendis nulla iste distinctio dolorum voluptas.Lorem ipsum dolor sit amet, consectetur adipisicing elit. <br>
Recusandae velit ducimus, consequatur id animi facilis dolor doloremque corrupti impedit odio cum, sed quam deserunt ipsam quia consequuntur perspiciatis repudiandae voluptatibus.

+ (5): TimaSmile, inco-media, MR_BaxtiyoR, ƒυтвσĻιѕт, TimX,

developer [323] [off]
#3 19:13:35, 9 May 2019

Davi_Uz, # Davi_Uz (09.05.2019 / 17:20)
parser haqida kim ma'lumot bera oladi??? lparser bu grabberning akasi garablangan malumotlarni o'zingizning bazangizga yozib borish yani malumotlardan to'liq kopiya olish.
grabberda esa shunchaki boshqa saytni sizni saytizda ko'rsatish holos

Davi_Uz [145] [off]
#4 08:38:39, 10 May 2019

developer, # developer (09.05.2019 / 19:13)
Davi_Uz, parser bu grabberning akasi garablangan malumotlarni o'zingizning bazangizga yozib borish yani malumotlardan to'liq kopiya olish.
grabberda esa shunchaki boshqa saytni sizni saytizda ko'rsatuni yaxshi bilaman.... (rahmat)
Parser buyruq kodlari bazaga yoxishga mysql koldlari shular haqida kim ma'lumot bera oladi?

developer [323] [off]
#5 10:08:49, 10 May 2019

Davi_Uz, # Davi_Uz (10.05.2019 / 08:38)
developer, uni yaxshi bilaman.... (rahmat)
Parser buyruq kodlari bazaga yoxishga mysql koldlari shular haqida kim ma'lumot bera oladi?me hali to'liq parser tuzishni o'rganganim yo'q lekin tushuncha bor grablangan sahifani to'liq keraksiz joylarini qirqib kerakli malumotlarni explode(); funksiyasi orqali massiv korinishiga ovolib insert orqali bazaga yozib olsa b'ladi.
bu oddiy matnli malumotlarni parser qilish uchun

sekin sekin o'rganamiz izlanib

developer [323] [off]
#11 22:05:13, 13 May 2019

programm011, # programm011 (13.05.2019 / 21:03)
preg_match('#<h1>(.*?)<h1>(.*?)<p>(.*?)</p>$#si',$uzg,$array);
$title = $array[1];
$text = $array[3];
mysql_query("INSERT INTO `table` SET `title` = '".$title."', `text` = '".$text."'");$arrayni qattan oldiz?

+ (1): code_one,

developer [323] [off]
#15 22:19:21, 29 May 2019

BestMaster, # BestMaster (28.05.2019 / 22:32)
developer, Pasta yozibdi namunanijohncms kutubxonasidagi maqolalarni bazaga yozib oladigan parser tuzish kerak
51 soniyadan keyin yozdi:
o'zim uni qanday yasash boyicha miyada algoritmi borku amalga qollay olmayapman
4 daqiqa 45 soniyadan keyin yozdi:

Запрос в MySQL-базу на создание таблицы для цитат
 
CREATE TABLE IF NOT EXISTS `bash` (
 
  `id` int(11) NOT NULL AUTO_INCREMENT,
 
  `quote_id` int(11) NOT NULL,
 
  `quote_text` mediumtext NOT NULL,
 
  PRIMARY KEY (`id`)
 
) ENGINE=MyISAM  DEFAULT CHARSET=utf8 AUTO_INCREMENT=1;
 
 
 
 
 
PHP-код парсинга
 
<?php
 
//Отключаем лимит ожидания и объявляем переменные
 
set_time_limit(0);
 
$content=NULL;$num_pages=NULL;
 
//Подключение к базе
 
//Подключаемся к базе данных
 
define ("HOST", "localhost");
 
define ("DATABASE", "ваша база данных");
 
define ("MYSQL_USER", "ваш юзер");
 
define ("MYSQL_PASS", "ваш пароль");
 
mysql_connect(HOST, MYSQL_USER, MYSQL_PASS);mysql_query("set names utf8");mysql_select_db(DATABASE);
 
//Узнаём кол-во страниц
 
$content=file_get_contents('http://bash.im/');
 
preg_match('|max="(.*?)"|is',$content,$num_pages);
 
//Парсим каждую страницу
 
for ($i=1; $i <= $num_pages[1]; $i++){$i2=false;
 
//Грузим страницы, вырезаем цитаты
 
$content=file_get_contents('http://bash.im/index/'.$i);
 
$quotes=explode('<span class="date">',$content);
 
foreach($quotes as $quote){
 
if($i2==false){$i2=true;continue;}
 
//Номер цитаты
 
preg_match('|class="id">#(.*?)</a>|is',$quote,$quote_id);
 
//Текст цитаты
 
preg_match('|<div class="text">(.*?)</div>|is',$quote,$quote_text);
 
//Конвертация текста в UTF-8 и экранирование символов
 
$quote_text=strip_tags(addslashes(htmlspecialchars(trim(iconv("windows-1251","UTF-8",$quote_text[1])))));
 
//Вписываем в базу
 
if($quote_text!=NULL){mysql_query("INSERT INTO `bash`(`id`,`quote_id`,`quote_text`) VALUES (NULL,'{$quote_id[1]}','{$quote_text}')");}
 
}
 
//От перенагрузки
 
sleep(2);
 
}
?>

shu kodni johncmsga moslash kerak
12 soat 14 daqiqa 16 soniyadan keyin yozdi:
sozni har hil belgilardan qanday tozalaydi qoshtirnoqlani orniga galati belgilar chiqib qolyapti

developer [323] [off]
#17 08:20:53, 1 Jun 2019

XuSha, # XuSha (30.05.2019 / 10:51)
developer, masalan deylik mp3 grabber tuzganda bosh sahifadagi kornishini grabladik keyin uni ichki korinishini grablap keyin ularni qanday qilib birlashtiramis, shu haqida batafsilroq malumot berolmimenimcha bitta grablasa boshqa saxifalar ham ishlayveradi
3 daqiqa 32 soniyadan keyin yozdi:
#15 shu parserni yasadim meni bazamga boshqa saytdagi malumotlar kopiya bob tushyapti faqat 30 40 ta malumot kopiya bob uyogiga toxtab qolyapti nima qilsa boladi?

+ (1): XuSha,

Mailzero [295] [off]
#20 20:54:14, 2 Oct 2019

john uchun yuklama grabi sinab ko`rilarchi:

<?php
define('_IN_JOHNCMS', 1);
$textl='Spaces';
require_once ("../incfiles/core.php");
require_once ("../incfiles/head.php");
header("Content-type:text/html; charset=utf-8");
$file = file_get_contents("http://alitest.ru/loads/spaces/?".$_SERVER['QUERY_STRING']);
echo $file;
require_once (".../incfiles/end.php");
?>