WARNING: All files in the current directory will be forcefully converted to UTF-8 while ignoring errors. However, they will all be stored in a separate utf8 folder. Please don’t use this script unless you know what you’re doing. I’m providing it for convenience. #!/usr/bin/python3 import os from os import listdir, Read More
”’ Separate text stream into hdf5 array format for far more efficient processing Memory-efficient Example: $ python3 text_to_hdf5.py /tmp/wikicomp-2014_arko.xml.bz2 “<articlePair id” –split-token-end “</articlePair>” ”’ import os import psutil import sys import h5py import gzip import bz2 import logging import argparse process = psutil.Process(os.getpid()) logging.basicConfig(format=’%(asctime)s : %(levelname)s : %(message)s’, \ level=logging.INFO) Read More
”’ Separate json stream(s) into hdf5 array format for far more efficient processing Example: $ python3 json_to_hdf5.py /tmp/jsonfile1.json.gz /tmp/jsonfile2.json.gz /tmp/jsonfile3.json.gz ”’ import os import json import psutil import sys import h5py import gzip import bz2 import logging import argparse process = psutil.Process(os.getpid()) logging.basicConfig(format=’%(asctime)s : %(levelname)s : %(message)s’, \ level=logging.INFO) logger Read More
Visualize Gensim word2vec model in TensorBoard. Output files can be uploaded to http://projector.tensorflow.org or by hosting TensorBoard code on your own web server and by using the projector json file output by the script. ”’ Output Little-Endian bytes and labels file from gensim model Also outputs necessary json config file portion Read More
Fix broken Korean filenames (e.g., Áö´ÉÇüÇüżҺм®±â.zip -> 지능형형태소분석기.zip) Fixes all filenames in current directory #!/usr/bin/python3 from os import listdir, rename from os.path import isfile, isdir, join, exists mypath = ‘.’ onlyfiles = [f for f in listdir(mypath) if isfile(join(mypath, f)) or isdir(join(mypath, f))] for f in onlyfiles: origF = f Read More
Please note that this tutorial (including the formatting) is a work in progress. I’m simply posting my progress in good faith. Your comments are welcomed. Introduction The news of Parsey McParseface achieving record-breaking parsing accuracy was interesting and thought-provoking. As far as I remember, it even hit some major public Read More