Blog

This experimental patch was developed primarily by analyzing FFMpeg and Chromium VP9 code and allows hardware acceleration of VP9 Profile 0 8-bit on Linux for supported NVIDIA cards (which seems to be common among YouTube videos). More details here: https://github.com/xtknight/vdpau-va-driver-vp9

Experimental code to convert sql.gz files full of SQL commands into usable CSV files, memory-efficiently.   ”’ Convert humongous compressed sql.gz files full of SQL commands into usable CSV files, memory-efficiently. ”’ import gzip import html import sys import string import unicodedata import pickle from html.parser import HTMLParser #gzfile = Read More

Train word embeddings using CoNLL corpus as input. Depends on: CoNLL Utils train_word_embeddings # take a CoNLL corpus and train word/doc embeddings import argparse import os import sys from conll_utils import * from gensim.models.word2vec import * # random from random import shuffle # necessary for seeing logs import logging logging.basicConfig(format=’%(asctime)s Read More

Module to help parse CoNLL (including optional special functionality for handling Korean characters). For use with POS tagging and dependency parsing. Depends on: Well-formed/projectivize filters conll_utils ”’ A set of classes to handle input and output of CoNLL-U files http://universaldependencies.org/docs/format.html The Parsed* classes are useful to store extra properties needed during Read More

Projectivize and well-formed filters translated to pure Python (translated from SyntaxNet code: document_filters.cc) ‘sentence’ object should be similar to that in SyntaxNet, with HEAD and other properties coming from the CoNLL file. See: CoNLL File Parsing Module for definition of the ‘sentence’ structure. Projectivize filter Check whether the given sentence is projective Read More

Convert Sejong POS-tagged corpus format to CoNLL-U format (useful for training Google SyntaxNet) #!/usr/bin/python3 # -*- coding: utf-8 -*- ”’ Convert Sejong POS-tagged corpus format to CoNLL-U format for use with Google SyntaxNet http://universaldependencies.org/docs/format.html Outputs training, testing, and tuning sets (60-20-20 ratio, randomly chosen) Arguments: ./corpus_to_conll.py <sejong_corpus_dir> <output_dir> sejong_corpus_dir should Read More

This is some code to interface with the ASUS ROG STRIX IMPACT Mouse on Linux. Attempts were made to perform DPI adjustment as well. These commands were inferred by using a USB packet sniffer on Windows. Please use at your own risk. This code may only work for the following Read More

This is a script for aligning English and Korean corpuses obtainable below. It also outputs vocab files for the NMT training TensorFlow example. https://wit3.fbk.eu/mono.php?release=XML_releases&tinfo=cleanedhtml_ted   #!/usr/bin/python3 import random from normalizer import Normalizer from tokenizer import Tokenizer ”’ Align TED English and Korean xml corpuses: https://wit3.fbk.eu/mono.php?release=XML_releases&tinfo=cleanedhtml_ted ”’ # character threshold # Read More

Auto-detect different Korean encodings in current directory (UTF-8, CP949, EUC-KR). Helpful when processing large amounts of Korean data with inconsistent encoding. Probably should be run recursively. #!/usr/bin/python3 import os from os import listdir, mkdir from os.path import isfile, join mypath = ‘.’ onlyfiles = [f for f in listdir(mypath) if Read More

This may help with processing of multiple large JSON files. It creates an indexed zip file containing all JSON files that can then be read with Python. ”’ Compress json stream(s) into indexed compressed format (zip file) for far more efficient processing Example: $ python3 json_to_zippack.py /tmp/jsonfile1.json /tmp/jsonfile2.json /tmp/jsonfile3.json ”’ Read More