권영기/web crawler - Revision history

Maintenance script: Repair batch-0004 pages from live compare

2026-03-27T00:37:15Z

Repair batch-0004 pages from live compare

← Older revision		Revision as of 00:37, 27 March 2026
Line 59:		Line 59:
	if pos is not -1 :		if pos is not -1 :
	for c in range(pos+1, len(line)) :		for c in range(pos+1, len(line)) :
	if line[c] is '"' :		if line[c] is '"' :
	fo2.write("\n")		fo2.write("\n")
	break		break
	fo2.write(line[c])		fo2.write(line[c])

	fo1.close()		fo1.close()
Line 76:		Line 76:
	fo = open("test2.html", "r")		fo = open("test2.html", "r")
	for line in fo.readlines():		for line in fo.readlines():
	urllib.urlretrieve(line,line.split('/')[-1])		urllib.urlretrieve(line,line.split('/')[-1])

	fo.close()		fo.close()
Line 85:		Line 85:

	line = 'http://cfile23.uf.tistory.com/original/2001D2044C945F80495C6F'		line = 'http://cfile23.uf.tistory.com/original/2001D2044C945F80495C6F'
	line.split('/')[[-1]] == '2001D2044C945F80495C6F'		line.split('/')-1 == '2001D2044C945F80495C6F'
	line.split('/')[[-2]] == 'original'		line.split('/')-2 == 'original'

	say = "This is a line of text"		say = "This is a line of text"
	part = line.split(' ')		part = line.split(' ')
	part == [['This', 'is', 'a', 'line', 'of', 'text']]		part == 'This', 'is', 'a', 'line', 'of', 'text'

	* swap		* swap
	Python 2.7.2+ (default, Oct 4 2011, 20:03:08)		Python 2.7.2+ (default, Oct 4 2011, 20:03:08)
	[GCC 4.6.1] on linux2		[GCC 4.6.1] on linux2
	Type "help", "copyright", "credits" or "license" for more information.		Type "help", "copyright", "credits" or "license" for more information.
	>>> first = 1		>>> first = 1
Line 111:		Line 111:

	* retrieve		* retrieve
	urllib.urlretrieve(url[, filename[, reporthook[[, data]]]])		urllib.urlretrieve(url[, filename[, reporthook, data]])
	http://docs.python.org/library/urllib.html		http://docs.python.org/library/urllib.html

Line 126:		Line 126:
	os.mkdir(mdir , 0755)		os.mkdir(mdir , 0755)

	type = ['/mp3', '/jpg', '/txt']		type = ['/mp3', '/jpg', '/txt']
	for t in type :		for t in type :
	create_dir(t)		create_dir(t)
Line 133:		Line 133:
	* os.getcwd() - Return a string representing the current working directory.		* os.getcwd() - Return a string representing the current working directory.
	* os.path.isdir(path) - Return True if path is an existing directory.		* os.path.isdir(path) - Return True if path is an existing directory.
	* os.mkdir(path[[, mode]]) - Create a directory named path with numeric mode mode. If the directory already exists, OSError is raised.		* os.mkdir(path, mode) - Create a directory named path with numeric mode mode. If the directory already exists, OSError is raised.

	http://docs.python.org/library/os.html		http://docs.python.org/library/os.html
Line 227:		Line 227:
	* 굳이 Python으로 GUI를 구현해야 할 필요성을 모르겠다.		* 굳이 Python으로 GUI를 구현해야 할 필요성을 모르겠다.
	----		----
	* 크롤러라니.. 음. -[[태진]]		* 크롤러라니.. 음. -태진

imported>trailblaze at 10:12, 15 August 2012

2012-08-15T10:12:56Z

← Older revision		Revision as of 10:12, 15 August 2012
Line 224:		Line 224:
	* wxpython		* wxpython
	* http://docs.python.org/tutorial/classes.html / 9.5까지.		* http://docs.python.org/tutorial/classes.html / 9.5까지.
			* 로그인이 가능한 페이지를 긁고 싶은데 방법을 모르겠다.
			* 굳이 Python으로 GUI를 구현해야 할 필요성을 모르겠다.
	----		----
	* 크롤러라니.. 음. -[[태진]]		* 크롤러라니.. 음. -[[태진]]

imported>jereneal20 at 12:31, 2 August 2012

2012-08-02T12:31:05Z

← Older revision		Revision as of 12:31, 2 August 2012
Line 224:		Line 224:
	* wxpython		* wxpython
	* http://docs.python.org/tutorial/classes.html / 9.5까지.		* http://docs.python.org/tutorial/classes.html / 9.5까지.

			----
			* 크롤러라니.. 음. -[[태진]]

imported>trailblaze at 07:28, 1 August 2012

2012-08-01T07:28:53Z

← Older revision		Revision as of 07:28, 1 August 2012
Line 198:		Line 198:
	http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10		http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10

	=== Eclipse + PyDev + wxPython ===		=== pywin32 ===

			* http://sourceforge.net/projects/pywin32/
			* 2.7버전.

			=== Eclipse + PyDev + wxPython + pywin32 ===

	# Eclipse 설치		# Eclipse 설치
Line 205:		Line 210:
	# Window > Preference > PyDev > Interpreter - Python > Auto Config		# Window > Preference > PyDev > Interpreter - Python > Auto Config
	# New Folder > /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode		# New Folder > /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode

			* 사실 라이브러리 다 깔아놓고 Auto Config하면 됨..

	== 개선해야 할 점 ==		== 개선해야 할 점 ==

imported>trailblaze at 01:54, 25 July 2012

2012-07-25T01:54:00Z

← Older revision		Revision as of 01:54, 25 July 2012
Line 197:		Line 197:
	sudo apt-get install gtk2-engines-pixbuf		sudo apt-get install gtk2-engines-pixbuf
	http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10		http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10

			=== Eclipse + PyDev + wxPython ===

			# Eclipse 설치
			# Eclipse에서, Help > Install New Software > Add > PyDev, Http://pydev.org/updates
			# 중간에 Aptana 신뢰하는지에 동의, 재시작
			# Window > Preference > PyDev > Interpreter - Python > Auto Config
			# New Folder > /usr/lib/python2.7/dist-packages/wx-2.8-gtk2-unicode

	== 개선해야 할 점 ==		== 개선해야 할 점 ==

imported>trailblaze at 08:43, 23 July 2012

2012-07-23T08:43:17Z

← Older revision		Revision as of 08:43, 23 July 2012
Line 208:		Line 208:
	* http://www.crummy.com/software/BeautifulSoup/ - 웹분석 라이브러리 BeautifulSoup 이용해보기		* http://www.crummy.com/software/BeautifulSoup/ - 웹분석 라이브러리 BeautifulSoup 이용해보기
	* wxpython		* wxpython
			* http://docs.python.org/tutorial/classes.html / 9.5까지.

imported>trailblaze at 05:47, 23 July 2012

2012-07-23T05:47:57Z

← Older revision		Revision as of 05:47, 23 July 2012
Line 192:		Line 192:
	* 설치 - apt-get install python-wxgtk2.8		* 설치 - apt-get install python-wxgtk2.8
	* 문서 - http://wiki.wxpython.org/How%20to%20Learn%20wxPython		* 문서 - http://wiki.wxpython.org/How%20to%20Learn%20wxPython

			* Gtk-WARNING **: 모듈을 module_path에서 찾을 수 없습니다: "pixmap"
			sudo apt-get install libgtkglextmm-x11-1.2-dev
			sudo apt-get install gtk2-engines-pixbuf
			http://stackoverflow.com/questions/8046667/vpython-error-on-ubuntu-11-10

	== 개선해야 할 점 ==		== 개선해야 할 점 ==

imported>trailblaze at 05:45, 23 July 2012

2012-07-23T05:45:19Z

← Older revision		Revision as of 05:45, 23 July 2012
Line 186:		Line 186:
	'http://comic.naver.com/webtoon/detail.nhn?titleId=449854&no=20&weekday=wed' 이면 럭키짱 20편이라는 소리.		'http://comic.naver.com/webtoon/detail.nhn?titleId=449854&no=20&weekday=wed' 이면 럭키짱 20편이라는 소리.

			=== wxPython ===

			자꾸 글자만 나오는 환경을 보니까 질리기 시작했다. 아직 완성은 멀었지만 GUI로 만들어보려고 함. 어차피 나중에 해야되니까...

			* 설치 - apt-get install python-wxgtk2.8
			* 문서 - http://wiki.wxpython.org/How%20to%20Learn%20wxPython

imported>trailblaze at 05:31, 23 July 2012

2012-07-23T05:31:17Z

← Older revision		Revision as of 05:31, 23 July 2012
Line 198:		Line 198:
	* http://wiki.kldp.org/wiki.php/SubversionBook/BranchingAndMerging		* http://wiki.kldp.org/wiki.php/SubversionBook/BranchingAndMerging
	* http://www.crummy.com/software/BeautifulSoup/ - 웹분석 라이브러리 BeautifulSoup 이용해보기		* http://www.crummy.com/software/BeautifulSoup/ - 웹분석 라이브러리 BeautifulSoup 이용해보기
			* wxpython

imported>trailblaze at 14:39, 21 July 2012

2012-07-21T14:39:06Z

← Older revision		Revision as of 14:39, 21 July 2012
Line 191:		Line 191:
	== 개선해야 할 점 ==		== 개선해야 할 점 ==

	* 파일을 저장할 떄 소스 파일에 저장이 되서 지저분하다.		* 파일을 저장할 떄 소스 파일에 저장이 되서 지저분하다. o
	* 필요하지 않은 파일까지 전부 긁어온다.		* 필요하지 않은 파일까지 전부 긁어온다. o
	* 내가 인터넷에서 jpg 파일 긁어오려고 만든 파이썬 코드 있는데 혹시 필요함? - [[서민관]]		* 내가 인터넷에서 jpg 파일 긁어오려고 만든 파이썬 코드 있는데 혹시 필요함? - [[서민관]]
	** 그러면 매우 감사하죠 ㅎㅎ - [[권영기]]		** 그러면 매우 감사하죠 ㅎㅎ - [[권영기]]
	* http://git-scm.com/		* http://git-scm.com/ - Git.
	* http://wiki.kldp.org/wiki.php/SubversionBook/BranchingAndMerging		* http://wiki.kldp.org/wiki.php/SubversionBook/BranchingAndMerging
			* http://www.crummy.com/software/BeautifulSoup/ - 웹분석 라이브러리 BeautifulSoup 이용해보기