pg – OracleBlog

PGAI和postgres.new的一些探索

2024-08-31 没有评论

有朋友提到最近最近比较热门的pg数据库和AI结合，提到了PGAI和postgres.new，看了相关资料。 PDF文件：pgai和postgres.new FreeMind文件：pgai和postgres.new.mm

一次大量事务日志生产的链条分析

2024-08-29 没有评论

数据传输工具，在全量同步中，往往是并发执行的，dts有个重写逻辑，是： 1）DTS碰到死锁的时候，会当前session中重试，继续写入数据。比如：DTS一批写入1024条数据，写到第1000条失败了，会在当前session中继续从头写入这1024条数据。这个过程可能会出现数据重复写入，会发生主键冲突 […]

..experience, Working case

sysbench报错max_prepared_stmt_count

2024-08-19 没有评论

在一个8c16g的阿里云mysql上进行sysbench，在run 压测的时候，报错max_prepared_stmt_count不够了：

sysbench oltp_read_write --mysql-host=rm-wedtteessttm.mysql.rds.aliyuncs.com --mysql-port=3306 --mysql-user=**** --mysql-password=********* --mysql-db=testdb --tables=30 --table-size=500000 --threads=64 --time=600 --report-interval=10 run

1	sysbench oltp_read_write --mysql-host=rm-wedtteessttm.mysql.rds.aliyuncs.com --mysql-port=3306 --mysql-user=** --mysql-password=******* --mysql-db=testdb --tables=30 --table-size=500000 --threads=64 --time=600 --report-interval=10 run

max_prepared_stmt_count的值，可以参考下面的表格进行设置：测试模型 p […]

..experience, Working case

谈谈向量数据库

2024-01-18 没有评论

最近公司里，向量数据库的需求越来越多。前几天请aws的诸位专家一起来开会讨论了一下，学到了不少知识。以下是我的心得和收获，仅代表我个人观点： 1. 知识库，图片识别，图片或视频近似度搜索，电商推荐等是很多公司的应用场景。 2. 向量计算，这其实每个数据库都可以拥有这个功能，向量近似度计算的方法：余弦 […]

..experience, Working case

pg_hba.conf文件的策略顺序

2022-02-11 没有评论

原来一直以为这个文件中，每一行的顺序是没有区别的，但是最近的一个需求，研究时发现，原来这个文件对登录的会话，是从上往下依次判断的：具体的规则是：

1. 登录的会话，会顺着pg_hba.conf的策略一条一条往下走。
2. 如果遇到匹配到的规则，那么做成判断，是信任，还是拒绝，还是验证密码。后续就不往下走继续判断规则了。
3. 如果某一条规则不符合，没匹配成功，就继续往下走，继续做判断，直到所有的规则都判断完毕。

1

2

3

1. 登录的会话，会顺着pg_hba.conf的策略一条一条往下走。

2. 如果遇到匹配到的规则，那么做成判断，是信任，还是拒绝，还是验证密码。后续就不往下走继续判断规则了。

3. 如果某一条规则不符合，没匹配成功，就继续往下走，继续做判断，直到所有的规则都判断完毕。

因此，如果我有一个用于备份的用户，叫anybackup（爱数备份软件），由于权限设置的 […]

..experience, Working case

减少mysqlbackup 对IO压力的影响

2021-12-20 没有评论

通常情况下，我们总是系统备份的速度是越快越好。但是，有时候这会导致IO压力，我们需要削峰处理。我之前写过一个减少rman备份对OLTP系统的影响，其实mysql的备份（使用 MySQL Enterprise Backup）也可以采用类似的方法：我们在mysqlbackup命令中，添加sleep […]

..experience, Working case

为什么说不要在pg 12前，用原生hash分区

2020-11-20 没有评论

pg在11之前，有原生的range分区和list分区，但是没有原生的hash分区。 hash分区是需要通过pg_pathman插件来实现。如果你从原来pg 10版本+pg_pathman实现hash分区，且分区数比较多，比如1024个分区，那么到你升级到pg 11之后，如果采用了原生分区，你会发现 […]

Study note

数据库内查询pg的表结构定义

2020-09-24 没有评论

需要利用到plperlu和自己写一个system函数。

-bash-4.2$ psql
psql (9.6.2)
Type "help" for help.

postgres=# create extension plperlu;
CREATE EXTENSION
postgres=# \dx
                      List of installed extensions
  Name   | Version |   Schema   |              Description               
---------+---------+------------+----------------------------------------
 plperlu | 1.0     | pg_catalog | PL/PerlU untrusted procedural language
 plpgsql | 1.0     | pg_catalog | PL/pgSQL procedural language
(2 rows)

1

2

3

4

5

6

7

8

9

10

11

12

13

-bash-4.2$ psql

psql (9.6.2)

Type "help" for help.

postgres=# create extension plperlu;

CREATE EXTENSION

postgres=# \dx

List of installed extensions

Name | Version | Schema | Description

---------+---------+------------+----------------------------------------

plperlu | 1.0 | pg_catalog | PL/PerlU untrusted procedural language

plpgsql | 1.0 | pg_catalog | PL/pgSQL procedural language

(2 rows)

postgres=# CREATE OR REPLACE FUNCTION system(text) RETURNS text
postgres-# AS 'my $cmd=shift; return `cd /tmp;$cmd`;' LANGUAGE plperlu;
CREATE FUNCTION
postgres=# 
postgres=#

1

2

3

4

5

postgres=# CREATE OR REPLACE FUNCTION system(text) RETURNS text

postgres-# AS 'my $cmd=shift; return `cd /tmp;$cmd`;' LANGUAGE plperlu;

CREATE FUNCTION

postgres=#

postgres=# select system('pg_dump -s -t orasup_test1 dbinfo2 |egrep -v "^--|^$"');
                       system                        
-----------------------------------------------------
 SET statement_timeout = 0;                         +
 SET lock_timeout = 0;                              +
 SET idle_in_transaction_session_timeout = 0;       +
 SET client_encoding = 'UTF8';                      +
 SET standard_conforming_strings = on;              +
 SET check_function_bodies = false;                 +
 SET client_min_messages = warning;                 +
 SET row_security = off;                            +
 SET search_path = public, pg_catalog;              +
 SET default_tablespace = '';                       +
 SET default_with_oids = false;                     +
 CREATE TABLE orasup_test1 (                        +
     a integer,                                     +
     b character varying(200)                       +
 );                                                 +
 ALTER TABLE orasup_test1 OWNER TO djidba_rw;       +
 CREATE INDEX idx_b ON orasup_test1 USING btree (b);+
 
(1 row)

postgres=# 
postgres=#

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

postgres=# select system('pg_dump -s -t orasup_test1 dbinfo2 |egrep -v "^--|^$"');

system

-----------------------------------------------------

SET statement_timeout = 0; +

SET lock_timeout = 0; +

SET idle_in_transaction_session_timeout = 0; +

SET client_encoding = 'UTF8'; +

SET standard_conforming_strings = on; +

SET check_function_bodies = false; +

SET client_min_messages = warning; +

SET row_security = off; +

SET search_path = public, pg_catalog; +

SET default_tablespace = ''; +

SET default_with_oids = false; +

CREATE TABLE orasup_test1 ( +

a integer, +

b character varying(200) +

); +

ALTER TABLE orasup_test1 OWNER TO djidba_rw; +

CREATE INDEX idx_b ON orasup_test1 USING btree (b);+

(1 row)

postgres=#

..experience, Working case

How to upgrade aws rds postgresql

2020-09-24 没有评论

## 0. Read the summary of upgrade pg https://docs.aws.amazon.com/zh_cn/AmazonRDS/latest/UserGuide/USER_UpgradeDBInstance.PostgreSQL.html ## 1. Prepare […]

Study note

pg常用大小查询

2020-09-24 没有评论

查出所有数据库大小：

select pg_database.datname, pg_database_size(pg_database.datname) AS size from pg_database order by size desc;

1	select pg_database.datname, pg_database_size(pg_database.datname) AS size from pg_database order by size desc;

查出所有表按大小排序并分离data与index：

SELECT
    table_name,
    pg_size_pretty(table_size) AS table_size,
    pg_size_pretty(indexes_size) AS indexes_size,
    pg_size_pretty(total_size) AS total_size
FROM (
    SELECT
        table_name,
        pg_table_size(table_name) AS table_size,
        pg_indexes_size(table_name) AS indexes_size,
        pg_total_relation_size(table_name) AS total_size
    FROM (
        SELECT ('"' || table_schema || '"."' || table_name || '"') AS table_name
        FROM information_schema.tables
    ) AS all_tables
    ORDER BY total_size DESC
) AS pretty_sizes;

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

SELECT

table_name,

pg_size_pretty(table_size) AS table_size,

pg_size_pretty(indexes_size) AS indexes_size,

pg_size_pretty(total_size) AS total_size

FROM (

SELECT

table_name,

pg_table_size(table_name) AS table_size,

pg_indexes_size(table_name) AS indexes_size,

pg_total_relation_size(table_name) AS total_size

FROM (

SELECT ('"' || table_schema || '"."' || table_name || '"') AS table_name

FROM information_schema.tables

) AS all_tables

ORDER BY total_size DESC

) AS pretty_sizes;

查看索引大小：

select relname as table_name,pg_size_pretty(pg_relation_size(relid)) as table_size, 
indexrelname as index_name,
pg_size_pretty(pg_relation_size(indexrelid)) as index_size from pg_stat_user_indexes 
order by pg_relation_size(indexrelid) desc;

1

2

3

4

select relname as table_name,pg_size_pretty(pg_relation_size(relid)) as table_size,

indexrelname as index_name,

pg_size_pretty(pg_relation_size(indexrelid)) as index_size from pg_stat_user_indexes

order by pg_relation_size(indexrelid) desc;

..experience, Working case

pg报错current transaction is aborted

2020-08-26 没有评论

在一个事务中，pg报错了current transaction is aborted：

mumy_corehrdban_psdb=> begin;
BEGIN
mumy_corehrdban_psdb=> select * from orasup_test1 ;
 a 
---
 1
 2
 3
(3 rows)

mumy_corehrdban_psdb=> insert into orasup_test1 values(111);
INSERT 0 1
mumy_corehrdban_psdb=> select * from not_exist;
ERROR:  relation "not_exist" does not exist
LINE 1: select * from not_exist;
                      ^
mumy_corehrdban_psdb=> insert into orasup_test1 values(222);
ERROR:  current transaction is aborted, commands ignored until end of transaction block

mumy_corehrdban_psdb=> select * from orasup_test1;
ERROR:  current transaction is aborted, commands ignored until end of transaction block
mumy_corehrdban_psdb=> 
mumy_corehrdban_psdb=> \d
ERROR:  current transaction is aborted, commands ignored until end of transaction block
mumy_corehrdban_psdb=> 
mumy_corehrdban_psdb=> 
mumy_corehrdban_psdb=> rollback;
ROLLBACK
mumy_corehrdban_psdb=> select * from orasup_test1;
 a 
---
 1
 2
 3
(3 rows)

mumy_corehrdban_psdb=>

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

mumy_corehrdban_psdb=> begin;

BEGIN

mumy_corehrdban_psdb=> select * from orasup_test1 ;

a

---

1

2

3

(3 rows)

mumy_corehrdban_psdb=> insert into orasup_test1 values(111);

INSERT 0 1

mumy_corehrdban_psdb=> select * from not_exist;

ERROR: relation "not_exist" does not exist

LINE 1: select * from not_exist;

^

mumy_corehrdban_psdb=> insert into orasup_test1 values(222);

ERROR: current transaction is aborted, commands ignored until end of transaction block

mumy_corehrdban_psdb=> select * from orasup_test1;

ERROR: current transaction is aborted, commands ignored until end of transaction block

mumy_corehrdban_psdb=>

mumy_corehrdban_psdb=> \d

ERROR: current transaction is aborted, commands ignored until end of transaction block

mumy_corehrdban_psdb=>

mumy_corehrdban_psdb=> rollback;

ROLLBACK

mumy_corehrdban_psdb=> select * from orasup_test1;

a

---

1

2

3

(3 rows)

mumy_corehrdban_psdb=>

原因是在一个事务中，pg如果遇到的Error的报错，会忽略后续的命令，后续所有命令都会报错：current transaction is abor […]

..experience, Working case

postgresql 检索当前数据库中所有的分区表

2020-05-15 没有评论

pg 10由于没有hash分区，而pg_pathman一直都是支持多种分区的。所以如果某些pg 11以前的系统，可能会混合部署pg原生分区和pg_pathman。要检查这种混合部署环境中的分区情况，可以用下面的sql：

select b.parent::text as part_table,
     'native partition' as part_tool,
CASE WHEN a.partstrat='r' THEN 'range'
     WHEN a.partstrat='l' THEN 'list'
     WHEN a.partstrat='h' THEN 'hash'
     ELSE 'other'
     END as part_type,
     b.cnt as part_cnt
from
pg_partitioned_table a,
(SELECT
    parent.oid,
    parent.relname          AS parent,
    COUNT(*) as cnt
FROM pg_inherits
    JOIN pg_class parent                 ON pg_inherits.inhparent = parent.oid
    JOIN pg_class child                  ON pg_inherits.inhrelid   = child.oid
    JOIN pg_namespace nmsp_parent        ON nmsp_parent.oid  = parent.relnamespace
    JOIN pg_namespace nmsp_child         ON nmsp_child.oid   = child.relnamespace
GROUP BY  oid,parent) b
where a.partrelid=b.oid
union all
select
     parent::text,
     'pg_pathman',
CASE WHEN parttype=1 THEN 'hash'
     WHEN parttype=2 THEN 'range'
     ELSE 'other'
END ,
count(*)
from pathman_partition_list
group by parent,parttype

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

select b.parent::text as part_table,

'native partition' as part_tool,

CASE WHEN a.partstrat='r' THEN 'range'

WHEN a.partstrat='l' THEN 'list'

WHEN a.partstrat='h' THEN 'hash'

ELSE 'other'

END as part_type,

b.cnt as part_cnt

from

pg_partitioned_table a,

(SELECT

parent.oid,

parent.relname AS parent,

COUNT(*) as cnt

FROM pg_inherits

JOIN pg_class parent ON pg_inherits.inhparent = parent.oid

JOIN pg_class child ON pg_inherits.inhrelid = child.oid

JOIN pg_namespace nmsp_parent ON nmsp_parent.oid = parent.relnamespace

JOIN pg_namespace nmsp_child ON nmsp_child.oid = child.relnamespace

GROUP BY oid,parent) b

where a.partrelid=b.oid

union all

select

parent::text,

'pg_pathman',

CASE WHEN parttype=1 THEN 'hash'

WHEN parttype=2 THEN 'range'

ELSE 'other'

END ,

count(*)

from pathman_partition_list

group by parent,parttype

显示结果如下 […]