'분류 전체보기'에 해당되는 글 96건

2017.12.05 [celery] How to run different multiple workers using different queue
2017.12.05 flask save session error
2017.11.09 How to rename metric column name in superset
2017.11.06 Creating python superset daemon using systemctl in centos
2017.11.03 Found unsuitable Qt version "" from NOTFOUND, this code requires Qt 4.x
2017.11.03 How to make game server log system
2017.11.03 Mysql visualization tool(그래프 툴) Super set!
2017.10.17 creating ubuntu python virtual env
2017.10.16 filebeat traceabck multiline config
2017.10.13 logstash conf example

[celery] How to run different multiple workers using different queue

celery 2017. 12. 5. 17:37

If you are going to run multiple workers and got some key error when doing async job in celery like (KeyError, Received unregistered task of type).
This would be a solution.

Key statement

"Use different Queue name and run worker with that Queue name". reference

0. Structure

folder/

tasks/

some_tasks.py

the_other_tasks.py

scheduler.py

1. scheduler.py

# import

from tasks.some_tasks import sum
from tasks.the_other_tasks import add

# use queue name, when you call task function
sum.apply_async(queue="some_tasks")
add.apply_async(queue="the_other_tasks")

2. tasks

A. some_tasks.py

app = Celery(..)
app.conf.task_default_queue = "some_tasks"

B. the_other_tasks.py

app = Celery(..)
app.conf.task_default_queue = "the_other_tasks"

3. running workers

$folder>celery -A tasks.some_tasks worker --loglevel=info --concurrency=1 -Q some_tasks

$folder>celery -A tasks.the_other_tasks worker --loglevel=info --concurrency=1 -Q the_other_tasks

*If you want to give name to worker, use -n option.
example) celery -A tasks.the_other_tasks worker --loglevel=info --concurrency=1 -n the_other_tasks -Q the_other_tasks

저작자표시

'celery' 카테고리의 다른 글

worker와 publisher가 init되는 것에 대한 오해가 있었다. (0)	2015.05.11
celery 오해와 진실 (0)	2015.03.13
Revoking task (0)	2014.11.24
예약 푸시 (0)	2014.11.20

시간을 거스르는자

ytkang86@gmail.com

flask save session error

python 2017. 12. 5. 10:58

Error

expires = pickle.load(f)
EOFError: Ran out of input

Solution

remove flask_session folder, and restart flask

ref

https://github.com/pallets/flask/issues/2216

저작자표시

'python' 카테고리의 다른 글

pytz localize vs normalize (0)	2018.02.26
python3 flask + pymongo + gevent, locust testing hangs (0)	2017.09.26
gevent monkey patch all, code location (0)	2017.09.14
flask with gevent monkey patch all (0)	2017.07.26
flask gevent spawn use a lot of memory (0)	2017.04.05

시간을 거스르는자

ytkang86@gmail.com

How to rename metric column name in superset

서버 교양 2017. 11. 9. 16:52

If you want to change each column by hands, follow this,

In your Slices,

1. select slice

2. click editing data resource button

3. click List Metrics

4. change Verbose name

Else if you want to change all column name, use sqlite,

superset uses sqlite to save their meta information and that file is located in ~/.superset/superset.db

open this file using sqlite GUI tool, and use replace function. replcae(field, 'origin', 'replacement')

example) update sql_metrics set verbose_name = replace(verbose_name, 'sum__', '') where table_id = 4

저작자표시

'서버 교양' 카테고리의 다른 글

filebeat -> logstash -> elasticsearch(AWS), filebeat timeout connection error (0)	2017.12.19
Creating python superset daemon using systemctl in centos (0)	2017.11.06
How to make game server log system (0)	2017.11.03
Mysql visualization tool(그래프 툴) Super set! (0)	2017.11.03
filebeat traceabck multiline config (0)	2017.10.16

시간을 거스르는자

ytkang86@gmail.com

Creating python superset daemon using systemctl in centos

서버 교양 2017. 11. 6. 17:51

1. make service file

sudo vim /etc/systemd/system/superset.service

fill this file with,

#superset.service

###########

[Unit]

Description=Visualization platform by Chrono

After=multi-user.target

[Service]

Type=simple

User=YOUR_USERNAME

ExecStart=/etc/init.d/superset

[Install]

WantedBy=default.target

2. make script

sudo vim /etc/init.d/superset

fill this file with,

#!/bin/bash

source /your/virtualenv/path/bin/activate

/your/virtualenv/path/bin/superset runserver

make script to executable

sudo chmod +x /etc/init.d/superset

3. update systemctl

sudo systemctl daemon-reload

4. start/stop/restart/status

sudo systemctl start superset

sudo systemctl stop superset

sudo systemctl restart superset

sudo systemctl status superset

저작자표시

'서버 교양' 카테고리의 다른 글

filebeat -> logstash -> elasticsearch(AWS), filebeat timeout connection error (0)	2017.12.19
How to rename metric column name in superset (0)	2017.11.09
How to make game server log system (0)	2017.11.03
Mysql visualization tool(그래프 툴) Super set! (0)	2017.11.03
filebeat traceabck multiline config (0)	2017.10.16

시간을 거스르는자

ytkang86@gmail.com

Found unsuitable Qt version "" from NOTFOUND, this code requires Qt 4.x

Android/OpenCV 2017. 11. 3. 22:38

If you are using mac,

Solution)

brew tap cartr/qt4
brew tap-pin cartr/qt4
brew install qt@4

저작자표시

'Android > OpenCV' 카테고리의 다른 글

OpenCV Manager app 필요 없이 OpenCV 사용하기 (1)	2014.07.03

시간을 거스르는자

ytkang86@gmail.com

How to make game server log system

서버 교양 2017. 11. 3. 18:55

1. Overview

2. Components

2-1. Log Creator: Game server instances -> F ilebeat
2-2. Log Distributor: Logstash (logstash config example)
2-3. Log Warehouse: s3
2-4. Log Analyzer: python schedule makes job, python celery workers do the tasks. (loading data from S3 via Athena and aggregate and save it to MySQL) you can monitor celery workers using flower
2-5. Log Visualizer: Elastic Search, Apache SuperSet

저작자표시

'서버 교양' 카테고리의 다른 글

How to rename metric column name in superset (0)	2017.11.09
Creating python superset daemon using systemctl in centos (0)	2017.11.06
Mysql visualization tool(그래프 툴) Super set! (0)	2017.11.03
filebeat traceabck multiline config (0)	2017.10.16
logstash conf example (0)	2017.10.13

시간을 거스르는자

ytkang86@gmail.com

Mysql visualization tool(그래프 툴) Super set!

서버 교양 2017. 11. 3. 17:28

아파치 그룹 오픈소스: https://github.com/apache/incubator-superset

아주 기가막힌 오픈 소스가 있었다! 이름도 수퍼한 superset!!

데이터 베이스만 연결하면 드레그엔 드랍만으로 멋진 통계그래프를 볼수 있다!

저작자표시

'서버 교양' 카테고리의 다른 글

Creating python superset daemon using systemctl in centos (0)	2017.11.06
How to make game server log system (0)	2017.11.03
filebeat traceabck multiline config (0)	2017.10.16
logstash conf example (0)	2017.10.13
python 3.5 flask gevent async requests test (0)	2017.04.07

시간을 거스르는자

ytkang86@gmail.com

creating ubuntu python virtual env

카테고리 없음 2017. 10. 17. 19:13

1. sudo apt-get update

2. sudo apt install python-pip

3. pip install --upgrade pip

4. pip install --upgrade virtualenv

5. virtualenv -p python3 p3.5

저작자표시

시간을 거스르는자

ytkang86@gmail.com

filebeat traceabck multiline config

서버 교양 2017. 10. 16. 15:41

multiline.pattern: '^[[:space:]]'

multiline.match: after

저작자표시

'서버 교양' 카테고리의 다른 글

How to make game server log system (0)	2017.11.03
Mysql visualization tool(그래프 툴) Super set! (0)	2017.11.03
logstash conf example (0)	2017.10.13
python 3.5 flask gevent async requests test (0)	2017.04.07
Logstash install (0)	2017.03.17

시간을 거스르는자

ytkang86@gmail.com

logstash conf example

서버 교양 2017. 10. 13. 16:33

# this is for logstash configuration
input {
  beats {
    port => 5044
  }
}

filter {
   grok {
   match => {
      "message" => "%{TIMESTAMP_ISO8601:timestamp} \[%{DATA:log_level}\] %{DATA:url} (?<content>(.)*)"
    }
   }

  if "_grokparsefailure" not in [tags] {
    if [content] {
      json {
        source => "content"
      }

      if [dest] == "mongo" {
        mutate {
          add_field => {"logType" => "query"}
        }
      }
      else {
        mutate {
          add_field => {"logType" => "log"}
        }
      }

    }

    mutate {
      remove_field => [ "content", "message", "tags"]
      gsub => ["timestamp", ",[0-9]+", ""]
    }
  }
  else {
    mutate {
      add_field => {"logType" => "else"}
    }
  }
}

output {
  if [logType] == "query" { # this is for query log
    mongodb {
      collection => "log"
      database => "gamedb"
      uri => "mongodb://"
      codec => json
      isodate => true
    }
  }
  else if [logType] == "log" and [log_level] == "INFO" {
    s3 {
      access_key_id => ""
      secret_access_key => ""
      region => "us-east-1"
      prefix => "%{+YYYY/MM/dd}/"
      bucket => ""
      size_file => 100000  # size: bytes,  100000 = 100kbytes
      time_file => 5
      codec => "json_lines"
    }
  }
  elasticsearch {
      hosts => [search.~~.es.aws.com:80]
  }
}

저작자표시

'서버 교양' 카테고리의 다른 글

Mysql visualization tool(그래프 툴) Super set! (0)	2017.11.03
filebeat traceabck multiline config (0)	2017.10.16
python 3.5 flask gevent async requests test (0)	2017.04.07
Logstash install (0)	2017.03.17
[펌] 인증 암호화와 해쉬 (0)	2016.10.29

시간을 거스르는자

ytkang86@gmail.com

'분류 전체보기'에 해당되는 글 96건

[celery] How to run different multiple workers using different queue

'celery' 카테고리의 다른 글

flask save session error

'python' 카테고리의 다른 글

How to rename metric column name in superset

'서버 교양' 카테고리의 다른 글

Creating python superset daemon using systemctl in centos

'서버 교양' 카테고리의 다른 글

Found unsuitable Qt version "" from NOTFOUND, this code requires Qt 4.x

'Android > OpenCV' 카테고리의 다른 글

How to make game server log system

'서버 교양' 카테고리의 다른 글

Mysql visualization tool(그래프 툴) Super set!

'서버 교양' 카테고리의 다른 글

creating ubuntu python virtual env

filebeat traceabck multiline config

'서버 교양' 카테고리의 다른 글

logstash conf example

'서버 교양' 카테고리의 다른 글

공지사항

카테고리

태그목록

글 보관함

달력

링크

시간을 거스르는자

LATEST FROM OUR BLOG

LATEST COMMENTS

BLOG VISITORS

티스토리툴바

« 2024/04 »
일	월	화	수	목	금	토
	1	2	3	4	5	6
7	8	9	10	11	12	13
14	15	16	17	18	19	20
21	22	23	24	25	26	27
28	29	30